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ABSTRACT 



This study examined factors related to survey response rate, 
particularly for teachers who participated in a nationwide survey. Using a 
newly developed statistical technique, the classification tree algorithm 
(CART) , this study classified the lowest response rate and highest response 
rate groups based on their school demographic characteristics. It also 
examined the differences in teacher satisfaction, decision making autonomy, 
and classroom climate between the two groups of responders. The teachers 
participated in the High Performance Learning Communities (HiPlaces) 
Assessment in 1996-1997, an assessment that was designed to examine the 
degree to which a broad range of recommendations for effective school reform 
were implemented in a school and the impact of these reform efforts. In 
1996-1997, 207 schools in 15 states participated in HiPlaces, and response 
data were available for 142 schools. The percentage of students eligible for 
free lunch appeared to be the most important factor among demographic 
variables in classifying schools with the lowest and highest response rates. 
In general, schools with the higher percentage of free lunch students have 
lower staff response rate, but the relationship is not linear. The CART 
approach allowed the examination of the nonlinear relationship. School size 
and the year the school joined the initiative were also important factors 
explaining the highest and lowest response rate groups. Systematic 
differences regarding the quality of the school as a work place and as an 
educational setting were found between the highest and lowest response rate 
groups. Teachers in the highest response rate group were more satisfied with 
their work, had less role conflict, and reported fewer barriers to 
implementation of the reform practices. The cumulative pattern of these 
findings suggests that response rates from staff tend to be lower in the 
schools that are of particular concern to educational reform efforts. 
(Contains 2 tables, 1 chart, and 10 references.) (SLD) 



Reproductions supplied by EDRS are the best that can be made 
from the original document. 



TM033162 



Q 



w 



PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL HAS 
BEEN GRANTED BY 

JVV^I^lvv 



U.S. DEPARTMENT OF EDUCATION 
Office of Educational Research and Improvement 
EDUCATIONAL RESOURCES INFORMATION 
J CENTER (ERIC) 

□ This document has been reproduced as 
received from the person or organization 
originating it. 



□ Minor changes have been made to 
improve reproduction quality. 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 

1 



• Points of view or opinions stated in this 
document do not necessarily represent 
official OERI position or policy. 



Factors for Teacher Response Rate in A Nationwide Middle Grades Survey 



By Minsuk Shim, Robert Felner, Eunjae Shim, Stephen Brand, Kenneth Gu 



School of Education 

National Center on Public Education and Social Policy 
University of Rhode Island 



Paper Presented at the Annual meeting of the American Educational Research Association 

in Montreal, Canada, 1999 




2 



BEST COPY AVAILABLE 



Introduction 



In various social science research, survey methods provide critical information regarding 
perceptions and opinions of the public. In educational research, survey research methodology is 
most often used to measure attitudes, feelings of teachers and students, and their perception of the 
school environment (Gay & Airasian, 2000). The accuracy of any results from survey data depends 
on who provides an answer to a particular question. In every survey, there are some people who 
agreed to respond but who do not answer all the questions. Those people who self-selected 
themselves not to provide any data at all are of great concern in any survey data collection (Fowler, 
1 993; Teston, 1998). Non-responses have two negative consequences for the quality of the estimates 
derived from the survey. First, non-responses can reduce the number of cases for whom data are 
available. As a result, the survey results would have less explanatory power. In many surveys where 
the sample size is fixed, researchers try to sample additional cases to compensate for those lost. 
Second, a more critical concern of non-responses is that they bias the sample, and make the sample 
systematically different from the population from which it was drawn (Sudman & Bradbum, 1984). 
This bias cannot be corrected with additional cases. The response rate, that is, the percentage of a 
sample who actually provides information is a basic parameter for evaluating a data collection effort. 
Researchers have provided many ways to reduce non-responses, from monetary incentives (James 
& Bolstein, 1990) to telephone follow-ups (Dillman, 1978). The primary purpose of the present 
study is to examine the factors that are related to response rate, especially for the teachers who 
participated in a nationwide survey. Using a newly developed statistical technique (classification 
tree algorithm), this study classifies the lowest response rate and highest response rate groups based 
on their school demographic characteristics. It also examines the differences in teacher satisfaction, 
decision making autonomy, and classroom climate between two groups as we believe these are the 
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factors that are related to overall feelings of well-being which is also related to higher response rate. 

Data Collection 

This study is particularly interested in response rate of teachers who participated in High 
Performance Learning Communities (HiPlaces) Assessment in 1996-97. Sampling bias is not a 
critical issue as this is not a nationally representative sample but a group of schools who participated 
in a nationwide educational reform initiative. HiPlaces Assessment is designed to examine the 
degree to which a broad range of recommendations for effective school reform are implemented in 
a school as well as to examine more fully their impact on students and staff. (Felner, Shim, Brand, 
Favazza, & Seitsinger, 2000). Nationally several reform initiatives employed the Hiplaces 
Assessment as an evaluation tool for their efforts. Among them, this study examines the response 
rate of teachers who participated in Middle Grade School State Policy Initiative (MGSSPI) 
sponsored by Carnegie Corporation of New York. A total of 207 schools in 15 states (Arkansas, 
California, Colorado, Connecticut, Delaware, Illinois, Maryland, Massachusetts, New Mexico, New 
York, North Dakota, Rhode Island, South Carolina, Texas, and Vermont) participated in HiPlaces 
Assessment in 1996-97. As many of the schools have participated in the survey for several years 
- the longest being started in 1993-94, there is a growing concern for lower response rate for the 
schools who have stayed in the initiative longer. In addition to Survey data, socio-demographic data 
from the National Public School Locator, a database maintained by the National Center for 
Education Statistics (NCES) are used in this study. The Locator carries school information and 
student and teacher data as of school year 1995-96, as reported to NCES by state education agency 
officials in each state. 

Method 

In this study, the primary method to classify the lowest response rate groups and highest 




response rate groups is classification tree algorithm known as CART (classification and regression 
tree; Breiman et al. 1984). When the typically more stringent theoretical and distributional 
assumptions (multivariate normality, common covariance matrix) of traditional methods are met, 
the traditional methods such as discriminant analysis and cluster analysis may be preferable method 
of classification. But as an explanatory technique, or as a technique when the assumptions of 
traditional methods are not met (more than often in real data), classification trees are unsurpassed 
in its accuracy. Our data clearly shows the violation of multivariate normality and common 
covariance matrix assumption (homogeneity). Among three of the most popular classification tree 
algorithms, CHAID (chi-square automatic interaction detection), CART (classification and 
regression tree) and QUEST (quick, unbiased, efficient, statistical tree), we used CART. CART 
algorithm is a procedure for analyzing categorical (classification) or continuous (regression) data. 
CART is a non-parametric procedure using exhaustive searches and computer-intensive testing to 
select the optional tree. It is typically more accurate for classifying new data than conventional 
stepwise procedures like linear regression, discriminant analysis, and logistic regression (Breiman 
et al.: 1984). Previous research has shown that CART is often is often 10 to 15 percent more 
accurate than parametric models. The program includes reliable estimates of error rates and is robust 
to outliers. Moreover, this method allows us to examine the interactions among several variables 
used in classification. 

The dependent (target) variable of this study is the response rate of teachers. The predictor 
variables are percentage of students who are eligible for free lunch, percentage of minority (non- 
white) students, school size (total number of students), urbanicity (rural, suburban, urban), network 
(year started participating in the initiative), building configuration (elementary and middle grades; 
middle grades only, middle and high school grades). These demographic variables were selected 
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based on previous research on survey non-responses (Monaco et. al.: 1997). The descriptive 
statistics of the sample is presented in Table 1 . 

We selected only the schools that have complete data on those variables. The resulting 
sample consists of 142 schools in 11 states. Average teacher response rate for those schools is 
76.46% with the standard deviation of 1 8.7. After we classify the lowest and highest response rate 
groups based on those factors, we examined the survey responses to see whether there are differences 
between two groups in terms of their report of job satisfaction, decision making, work and classroom 
climate, and barriers to implementation using t-test. When teachers are more satisfied with their 
work and initiative, they would have more positive attitudes towards the survey. 

Results 

Among the predictor variables that were used in the model, percentage of students eligible 
for free lunch was selected for the first level tree (see Chart 1). School in which 26% or more of the 
students were eligible for free lunch showed significantly lower response rate than those with smaller 
proportion of free lunch students (73.81% compared to 83.49%). For the higher response rate 
groups, percent free lunch (10% or higher) and school size (with less than 480 students) significantly 
separate the response rate groups. That is, relatively small size schools with percentage of free lunch 
between 10% and 26% have the highest response rate among all schools (94.83% - 6 schools). On 
the other hand, for the lowest response rate groups, percent free lunch students and network appeared 
to be significant factors. As we hypothesized, schools which participated in the survey 2 years or 
more had significantly lower response rate than those participated in the survey for the first time in 
1996-97. The lowest response rate group appears to be the schools that participated in the survey 
2 years or more with percentage of free lunch students between 60% and 73% (58.67% - 1 5 schools). 

In general, teachers at schools with higher percentage of students eligible for free lunch have 
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lower response rate. However, one of the interesting findings of this study is that this relationship 
is not always linear. Teachers at 8 schools with less than 10% free lunch students have lower 
response rate that those at 13 schools with more than 73% free lunch students (71.25% compared 
to 82.38%). Again, 13 schools with more than 73% free lunch students have higher staff response 
rate than 15 schools with percentage free lunch students between 60% and 73% (82.38% compared 
to 58.67%). School size was a significant factor for explaining higher response groups only and 
network was a significant factor for explaining lower response rate groups only. 

In order to better understand the linkage between the demographic characteristics of the 
school and staff response rates, we next examined staff ratings of their work environment in the 
lowest and the highest response rate groups (see Table 2). Systematic differences were found 
between these groups in ratings of job satisfaction, work and classroom climate. Teachers in the 
highest response rate group have significantly higher ratings of their job satisfaction regarding 
students, parents, resources and overall although their satisfaction with the pay was lower than those 
in the lowest response rate group. They reported higher decision making autonomy, less role 
conflict, more positive and less disruptive behaviors of students in their classroom. They also rated 
their work climate more highly than those in the lowest response rate group. Finally, they reported 
less barriers to implementation of middle grades practice. 

Conclusions and Implementation 

In classifying schools with lowest and highest response rates, percentage of students eligible 
for free lunch appears to be the most important factor among other demographic variables. In 
general, schools with higher percentage of free lunch students have lower staff response rate. 
However, the relationship is not linear. CART allows us to examine the non-linear (interaction) 
relationship between response rate and socio-demographic characteristics of the schools. Among 
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other demographic variables, school size and year they joined the initiative were important factors 
for explaining the highest and the lowest response rate groups. Systematic differences regarding the 
quality of the school as a work place and as an educational setting were found between the highest 
and the lowest response rate groups. Teachers in the highest response rate group are more satisfied 
with their work, have less role conflict, and report less barriers to implementation of the reform 
practices. 

The cumulative pattern of these findings suggest that response rates from staff tend to be 
lower in those schools that are of particular concern to educational reform efforts; i.e., high poverty 
and large schools. The relatively low response rates in those schools may reflect, in part, work 
conditions in which teachers are demoralized and dissatisfied. In order to increase response rates 
from staff in these conditions, it may be particularly important to cultivate conditions under which 
staff “buy in” to the survey process and the underlying school reform initiative under which the 
survey is conducted. In the absence of such buy-in, low response rate may yield less complete 
information on practices and attitudes in low-income school settings. 
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Table 1. Descriptive Statistics of the Sample (N=142) 
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Chart 1 



SFRATE 



Mean 


76.46 


Std.Dev 


18.70 


n 


142 (100.00%) 


Predicted 


76.46 



PFRLUNCH 
Improvements 8.6723 
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>25.914 



Mean 
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39 (27.46%) 


Predicted 


83.49 
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Improvements 0.61 37 
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Mean 
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103 (72.54%) 


Predicted 


73.81 
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NETWORK 

Improvements 13.3968 
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Mean 
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8 (5.63%) 
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Mean 


86.65 


Std.Dev 
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Mean 
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Mean 
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Predicted 


82,84 
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Table 2. Differences between schools with highest and lowest staff response rates 

(MANOVA) 



Variables 


Mean 

difference 


F 


P 


Job satisfaction 


Extrinsic rewards 


-.268 


9.302 


.002 


Intrinsic rewards 


.057 


.027 


.869 


Instructional resources 


.431 


16.806 


.000 


Colleagues 


.085 


1.233 


.268 


Student behavior 


.293 


9.674 


.002 


Parent/community support 


.806 


76.801 


.000 


Job evaluation feedback 


.086 


.296 


.587 


Building administration 


.095 


.988 


.321 


Inputs in decision making 


.092 


.562 


.454 


Overall job satisfaction 


.206 


10.107 


.002 


Decision 

making 


Participation in decision making 


.214 


4.602 


.033 


Centralization in decision making 


-.070 


.864 


.353 


Decision making authority regarding 
intra- team(grade) practices 


.211 


9.024 


.003 


Decision making authority concerning 
school-wide policies and practices 


.232 


12.952 


.000 


Overall team/grade decision making 


.157 


7.944 


.005 


Classroom 

climate 


Positive peer interaction 


.178 


5.810 


.016 


Disruptiveness 


-.151 


3.872 


.049 


Positive interaction between teachers and 
students 


.007 


.010 


.919 


Academic orientation 


.230 


7.368 


.007 


Overall positive classroom climate 


.147 


6.952 


.009 


Barriers to 
implementation 
of reform 
initiatives 


School and teacher readiness 


-.289 


15.238 


.000 


Parental concerns 


-.150 


5.868 


.016 


Lack of external formal organizational 
support 


-.322 


14.044 


.000 


Overall barriers to implementation 


-.235 


17.442 


.000 




TM033162 

-4 

® 

U.S. Department of Education 

Office of Educational Research and Improvement (OERI) 

National Library of Education (NLE) 

Educational Resources Information Center (ERIC) 

REPRODUCTION RELEASE 



(Specific Document) 

I. DOCUMENT IDENTIFICATION: 



Title ' Factors for Teacher Response Rate in A Nationwide Middle 


Grades Survey 


Author(s): Mins uk Shim/ Robert Felner, Euniae Shim , Stephen Brand, & Kenneth Gu 


Corporate Source: Paper presented at the Annual Meeting of 


Publication Date: 


the American Educational Research Association 


April 1999 


(Montreal, Canada, 1999) 







II. REPRODUCTION RELEASE: 

In order to disseminate as widely as possible timely and significant materials of interest to the educational community, documents announced 
in the monthly abstract journal of the ERIC system, Resources in Education (RIE), are usually made available to users in microfiche, reproduced 
paper copy, and electronic media, and sold through the ERIC Document Reproduction Service (EDRS). Credit is given to the source of each 
document, and, if reproduction release is granted, one of the following notices is affixed to the document. 



If permission is granted to reproduce and disseminate the identified document, please CHECK ONE of the following three options and sign 
at the bottom of the page. 

The sample sticker shown below will be The sample sticker shown below will be The sample sticker shown below will be 



affixed to all Level 1 documents affixed to all Level 2A documents affixed to all Level 2B documents 



PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL HAS 
BEEN GRANTED BY 




PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL IN 
MICROFICHE, AND IN ELECTRONIC MEDIA 
FOR ERIC COLLECTION SUBSCRIBERS ONLY, 
HAS BEEN GRANTED BY 




PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL IN 
MICROFICHE ONLY HAS BEEN GRANTED BY 












TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 




TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 




TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 


1 




2A 




2B 



Level 1 


Level 2A 


Level 2B 


t 


T 


t 


□ 

□ 

□ 


Check here for Level 1 release, permitting 
reproduction and dissemination in microfiche or other 
ERIC archival media (e.g., electronic) and paper copy. 


Check here for Level 2A release, permitting reproduction 
and dissemination in microfiche and in electronic media 
for ERIC archival collection subscribers only 


Check here for Level 2B release, permitting 
reproduction and dissemination in microfiche only 



Documents will be processed as indicated provided reproduction quality permits. 

If permission to reproduce is granted, but no box is checked, documents will be processed at Level 1. 



Sign 

here,-* 

please 



I hereby grant to the Educational Resources Information Center (ERIC) nonexclusive permission to reproduce and disseminate this 
document as indicated above. Reproduction from the ERIC microfiche or electronic media by persons other than ERIC employees and 
its system contractors requires permission from the copyright holder. Exception is made for non-profit reproduction by libraries and 
other service agencies to satisfy information needs of educators in response to discrete inquiries. 



Signature: 










Organization/Address: University Qf Rhode Island 



Printed Name/Position/Title: 



Minsuk Shim/ Ph.D. 



Telephone: 



874-4162 



FAX: 

(4fVn S 74-5453 



o 

ERIC 




M 


E-Mail Address: 


Date: 


% 


mshim0uri.edu 


Mav 15,2001 



III. DOCUMENT AVAILABILITY INFORMATION (FROM NON-ERIC SOURCE): 

If permission to reproduce is not granted to ERIC, or, if you wish ERIC to cite the availability of the document from another source, please 
provide the following information regarding the availability of the document. (ERIC will not announce a document unless it is publicly available, 
and a dependable source can be specified. Contributors should also be aware that ERIC selection criteria are significantly more stringent for 
documents that cannot be made available through EDRS.) 



Publisher/Distributor: 



Address: 



Price: 



IV. REFERRAL OF ERIC TO COPYRIGHT/REPRODUCTION RIGHTS HOLDER: 

If the right to grant this reproduction release is held by someone other than the addressee, please provide the appropriate name and address: 




V. WHERE TO SEND THIS FORM: 



Send this form to the following ERIC Clearinghouse: 



However, if solicited by the ERIC Facility, or if making an unsolicited contribution to ERIC, return this form (and the document being contributed) 
to: 



EFF-088 (Rev. 2/2000) 




ERIC Processing and Reference Facility 
4483-A Forbes Boulevard 
Lanham, Maryland 20706 

Telephone: 301-552-4200 
Toll Free: 800-799-3742 
FAX: 301-552-4700 
e-mail: ericfac@inet.ed.gov 
WWW: http://ericfac.piccard.csc.com 



